Minimum mean square estimation and neural networks

Neural networks for estimation, such as the multilayer perceptron (MLP) and functional link net (FLN), are shown to approximate the minimum mean square estimator rather than the maximum likelihood estimator or others. Cramer-Rao maximum a posteriori lower bounds on estimation error can therefore be used to approximately bound network training error, when a statistical signal model is available for its inputs and the desired outputs are Gaussian. The bounds help the user to determine when to stop training, and to determine how close to optimal the neural net’s performance is. When a linear preprocessor is sought to compress raw data, before it is input into a neural network, the bounds can be used to determine the relative optimality of several candidate linear preprocessors or transforms. A method is proposed for re-ordering the rows of the prepr ocessor’s transform matrix. It is shown that a single linear transformation can be used, even when more than one parameter is estimated by the network. Published in : Neurocomputing, vol. 13, September 1996, pp. 59-74.